Techniques for enabling or establishing the use of face recognition algorithms

ABSTRACT

Embodiments described herein facilitate or enhance the implementation of image recognition processes which can perform recognition on images to identify objects and/or faces by class or by people.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/962,494, filed Dec. 7, 2010, now U.S. Pat. No. 8,630,493, which is adivisional of U.S. patent application Ser. No. 11/685,106, filed Mar.12, 2007, now U.S. Pat. No. 8,571,272, which claims benefit of priorityto U.S. Provisional Patent Application Nos. 60/781,691, 60/781,690 and60/781,966, all filed Mar. 12, 2006. All of the aforementioned priorityapplications are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to the field of digital imageprocessing. More particularly, the disclosed embodiments relate to asystem and method for enabling the use of captured images.

BACKGROUND

Digital photography has become a consumer application of greatsignificance. It has afforded individuals convenience in capturing andsharing digital images. Devices that capture digital images have becomelow-cost, and the ability to send pictures from one location to theother has been one of the driving forces in the drive for more networkbandwidth.

Due to the relative low cost of memory and the availability of devicesand platforms from which digital images can be viewed, the averageconsumer maintains most digital images on computer-readable mediums,such as hard drives, CD-ROMs, and flash memory. The use of file foldersis the primary source of organization, although applications have beencreated to aid users in organizing and viewing digital images. Somesearch engines, such as GOOGLE, also enables users to search for images,primarily by matching text-based search input to text metadata orcontent associated with images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a system for generating synthesized images toaccommodate or compensate for pose and/or occlusion, according anembodiment of the invention.

FIG. 1B and FIG. 1C illustrate how a digital photograph may besynthesized, according to one or more embodiments of the invention.

FIG. 2 illustrates a method for generating synthesized images toaccommodate or compensate for pose and/or occlusion, according anembodiment of the invention.

FIG. 3 illustrates a method for determining a set of training images forperforming image recognition on one or more objects of interest, underan embodiment of the invention.

FIG. 4 illustrates a semi-supervised process for dynamically altering anacceptance threshold for object recognition, according to one or moreembodiments of the invention.

FIG. 5 is a graph showing a performance of one embodiment describedherein, on multiple datasets.

FIG. 6 illustrates a method for determining face or object registration,according to an embodiment of the invention.

FIG. 7 illustrates an image recognition system constructed according toembodiments described herein.

FIG. 8 illustrates a computer system on which embodiments such asdescribed with FIGS. 1-7, and combinations thereof, may be implemented

DETAILED DESCRIPTION

Embodiments described herein facilitate or enhance the implementation ofimage recognition processes which can perform recognition on images toidentify objects and/or faces by class or by people.

When employed on a collection of digital images, embodiments describedherein enable the use of several preliminary steps to enhance theperformance of recognition and the results derived therein. Thesepreliminary steps may include (i) establishing a labeled training set ofimages from which subsequent recognition can take place; (ii)registering faces or objects, to facilitate the ability of a recognitionalgorithm to be able to determine corresponding points on differentobjects or faces; and (iii) enabling the recognition algorithms tocompensate or otherwise overcome occlusion or posing of the object thatwould otherwise diminish or preclude the algorithm from detectingfeatures or corresponding points of the object of interest.

Embodiments described herein provide for computer-implemented imagerecognition of objects in images. According to an embodiment, an objectis identified from an image, where the object that is posed so as toocclude one or more regions of the object in the image. Image data issynthesized for at least some of the one or more occluded regions of theobject. Recognition is performed on the object using the synthesizedimage data.

According to another embodiment, recognition is facilitated or enabledthrough the creation of a training set, using a programmatic andadaptive technique for identifying or selecting images for use in atraining set. In one embodiment, a set of candidate images areprogrammatically selected, from a collection of images, for a trainingset. The set of images are then presented to the user. Individual imagesfrom the candidate set are presented to the user for feedback or input,and select images presented to the user may be identified as part of thetraining set.

According to an embodiment such as described, the training set maycontinuously be modified or updated. As an addition or alternative, athreshold criteria for selecting images in the training set may be madeadaptive based on factors that include the error rate or other measureof how well recognition is performed with the given training set.

Examples of objects that may be used with one or more embodimentsinclude a face, a person, apparel, clothing, merchandise, and animals.

According to another embodiment, image recognition includes determininga region of an image where an object that is to be recognized is likelylocated. The object of interest may be known to have specific set offeatures, which may be defined in variety of ways (e.g. geometricconstraints) through two or three dimensional models. For each of aplurality of features of the object of interest, a set of hypothesesregions of the image may be determined. These hypotheses regions maycorrespond to likely locations of that feature in the object that is tobe recognized. A most likely location of at least some of the pluralityof features using may then be determined using the set of one or morehypothesis regions for all of the plurality of features.

Various applications and implementations are contemplated for one ormore embodiments of the invention.

As used herein, the term “image data” is intended to mean data thatcorresponds to or is based on discrete portions of a captured image. Forexample, with digital images, such as those provided in a JPEG format,the image data may correspond to data or information about pixels thatform the image, or data or information determined from pixels of theimage.

The terms “recognize”, or “recognition”, or variants thereof, in thecontext of an image or image data (e.g. “recognize an image”) is meantto means that a determination is made as to what the image correlatesto, represents, identifies, means, and/or a context provided by theimage. Recognition does not mean a determination of identity by name,unless stated so expressly, as name identification may require anadditional step of correlation.

As used herein, the terms “programmatic”, “programmatically” orvariations thereof mean through execution of code, programming or otherlogic. A programmatic action may be performed with software, firmware orhardware, and generally without user-intervention, albeit notnecessarily automatically, as the action may be manually triggered.

One or more embodiments described herein may be implemented usingprogrammatic elements, often referred to as modules or components,although other names may be used. Such programmatic elements may includea program, a subroutine, a portion of a program, or a software componentor a hardware component capable of performing one or more stated tasksor functions. As used herein, a module or component, can exist on ahardware component independently of other modules/components or amodule/component can be a shared element or process of othermodules/components, programs or machines. A module or component mayreside on one machine, such as on a client or on a server, or amodule/component may be distributed amongst multiple machines, such ason multiple clients or server machines. Any system described may beimplemented in whole or in part on a server, or as part of a networkservice. Alternatively, a system such as described herein may beimplemented on a local computer or terminal, in whole or in part. Ineither case, implementation of system provided for in this applicationmay require use of memory, processors and network resources (includingdata ports, and signal lines (optical, electrical etc.), unless statedotherwise.

Embodiments described herein generally require the use of computers,including processing and memory resources. For example, systemsdescribed herein may be implemented on a server or network service. Suchservers may connect and be used by users over networks such as theInternet, or by a combination of networks, such as cellular networks andthe Internet. Alternatively, one or more embodiments described hereinmay be implemented locally, in whole or in part, on computing machinessuch as desktops, cellular phones, personal digital assistances orlaptop computers. Thus, memory, processing and network resources may allbe used in connection with the establishment, use or performance of anyembodiment described herein (including with the performance of anymethod or with the implementation of any system).

Furthermore, one or more embodiments described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown in figures below provide examples of processingresources and computer-readable mediums on which instructions forimplementing embodiments of the invention can be carried and/orexecuted. In particular, the numerous machines shown with embodiments ofthe invention include processor(s) and various forms of memory forholding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as carried on many cell phones and personal digitalassistants (PDAs)), and magnetic memory. Computers, terminals, networkenabled devices (e.g. mobile devices such as cell phones) are allexamples of machines and devices that utilize processors, memory, andinstructions stored on computer-readable mediums.

Recognition of Occluded Portions of an Object in an Image

In the field of digital photography, objects in images are often posedat an angle or tilt with respect to the viewpoint of the camera lens.The ability of image recognition systems or software to work properly onsuch objects may be negatively affected. For example, an object at atilt or angle may have portions of the object occluded or otherwise notvisible in the image. Embodiments described herein accommodate orotherwise compensate for occurrences of tilts in the pose of objects,including in instances when occlusion is present in the object ofinterest, when recognition is performed on posed objects of images.

Facial recognition is an example of one type of recognition performedwith digitally captured images. In general, facial recognitionidentifies the location of a face of a person in an image, then seeks touse a signature of the person's face to identify that person by name orby association with other images that contain that person. Non-facialrecognition may be similarly performed, in that a pre-determined objectmay be detected from a digital image, then a signature of the object maybe used to identify the object by class or otherwise.

One or more embodiments recognize that for image recognition to beperformed accurately on, for example, consumer images or randomly posedobjects in images, either (i) the face/object being recognized can beassumed to be frontally aligned, or (ii) occlusion or pose distortionsof the object of interest must be accommodated or compensated forprogrammatically or otherwise. For example, in the absence of occlusioncompensation, facial or object image recognition may require utilizationof simplifying assumptions about the face or object being recognized.Such assumptions may assume, for example, the face or object is in afrontal position (relative to the camera) when the image is captured.

In practice, large variations in viewpoints are extremely common inconsumer photographs. With regard to facial recognition, in particular,head rotations can produce significant changes and distortions in thefacial appearance of an individual in a photograph. Such effectsgenerally cause significant degradation of face recognition performance.

Embodiments described herein enable generation of a realistic image of aface or other object, using as input, an original image that has one ormore portions of the face or object occluded or rotated. In particular,one or more embodiments provide that a synthesized image may begenerated of a face or object. The synthesized image may present theface or object in a frontal view and at a specified position orviewpoint. The input for creating the synthesized image may correspondto an image that contains the face or object posed at some arbitraryviewpoint that causes occlusion.

In one embodiment, a synthesized frontal view of a face can replace theoriginal photograph as input in a face recognition system in order toproduce more accurate recognition performance. Alternatively, asynthesized frontal view of an object can replace the originalphotograph as input in an object recognition system. For example, theobject may correspond to clothes, apparel, merchandise, a person (ratherthan just a face) or even an animal or animal face.

In another embodiment, a combination of the two-dimensional originalview and a synthesized frontal view can be used at the same time. Forthis task, while matching two faces, either a weighted average of thetwo, or the minimum distance of the two metrics can be used.

One or more embodiments described herein recognize that a frontal viewof an object may be synthesized from a non-frontally viewed image ofthat same object using comparison or prior knowledge of a genericappearance of the object or of a class or kind of the object. In oneembodiment, a frontal view of a face may be synthesized from anon-frontally viewed image of that face using prior knowledge of ageneric appearance of a face (or a type of face) or an object. Accordingto one embodiment, a geometric three-dimensional model of an object(including face) that is to be recognized is used to provide priorknowledge or reference for synthesis.

FIG. 1A illustrates a system for generating synthesized images toaccommodate or compensate for pose and/or occlusion, according anembodiment of the invention. In FIG. 1A, a system 100 may comprise aplurality of modules include an object detector 110, a pose or occlusiondetection module 120, and a synthesizer 130.

System 100 may receive image data 112 from an image source 102. Examplesof the image source 102 include a digitally captured image, such asimages captured by digital cameras or images presented with othercontent. The resolution of the image source 102 may vary, depending on,for example, an application of an embodiment of FIG. 1A, as well asother parameters.

The object detector 110 may scan image data 112, corresponding to theimage file 102, to detect an object of interest. In one embodiment, theobject of interest corresponds to a face of a person. Detection of theface may be performed by scanning the image data 112 for features thatare indicative of a face, such as the eyes, eyebrows, or corners of amouth. Other examples of objects of interest include clothes, apparel,merchandise, a person (rather than just a face), an animal or animalface. For each type of object that is of interest for recognition,markers of that object's class may be pre-defined, and images may bescanned for such markers in order to detect presence of those objects inan image.

Object image data 122 may be used by the pose/ occlusion module 120 todetermine whether the detected image is posed to a degree that wouldcause occlusion or other significant deterioration of recognition.Alternatively, the pose/occlusion module 120 may detect, from the objectimage data 122, whether the image is posed to a degree that would hinderor interfere image recognition analysis on the object.

The pose/occlusion module 120 may determine that the pose of the objectof interest within an acceptable threshold, in which case no synthesisis performed. If, however, the pose/occlusion module 120 determines thatthe pose of the object of interest is significant, or if there isocclusion from the pose of the object of interest, the pose/occlusionmodule 120 executes to determine augmentation information 132. In oneembodiment, augmentation information 132 may utilize a three-dimensionalmodel of a generic or non-specific object (of the same class as theobject of interest) to quantify the pose (e.g. orientation) and positionof the object of interest, as well as the intensity of discrete pointsof the object of interest.

The synthesizer 130 may utilize the object image data 122, as well asthe augmentation information 132 as determined from the pose/occlusionmodule 120, to generate a synthesized image 142 of the image file 102.The synthesized image 142 may use the augmentation information 132 tore-orient and possibly re-position the object of interest. For example,if the object of interest is a face that is tilted one way or another,the synthesized image 142 may re-align the face so as to orient ittowards a hypothetical camera angle.

In an embodiment, synthesized image 142 is simply an output of system100. For example, images in a photo-library may be scanned, and faces ofpersons appearing at an angle may be synthesized and presented as anindependent set of images, each with the images of the person providedfrontally. In another embodiment, synthesized image 142 is generated forpurpose of performing a process such as image recognition. In cases whenthe object of interest, as determined from the image file 102 is tooangled and/or occluded to perform recognition, synthesized image 142 maybe generated as replacement input. Image recognition module 140 mayperform recognition on the object of interest using the synthesizedimage 142 as input.

FIG. 1B and FIG. 1C illustrate how a digital photograph may besynthesized, according to one or more embodiments of the invention. FIG.1B illustrates a side view of a face 160, as well as a synthesizedfrontal view 170 generated by a system such as described with anembodiment of FIG. 1. In an embodiment, a synthesis of frontal viewfaces from non-frontal faces in photographs can be used as apreprocessing step in view-dependent face recognition systems. Thus,embodiments described herein enable improved recognition performance byallowing face recognition systems to compare faces perfectly aligned andin standard pose, where appearance variations due to changes inviewpoints are eliminated.

One or more embodiments described herein recognize that a frontal viewof an object may be synthesized from a non-frontally viewed image ofthat same object using comparison or prior knowledge of a genericappearance of the object or of a class or kind of the object. FIG. 2illustrates a method for generating synthesized images to accommodate orcompensate for pose and/or occlusion, according an embodiment of theinvention. A frontal view of a face may be synthesized from anon-frontally viewed image of that face using prior knowledge of ageneric appearance of a face (or a type of face). A method such asdescribed by an embodiment of FIG. 2 may be performed using a systemsuch as described with an embodiment of FIG. 1A, and further implementedusing a computer such as described with FIG. 8.

Step 210 provides that a geometric three-dimensional model of an objectthat is to be recognized is formulated. The model may be generic to aclass of the object. For example, in face recognition, the model may befor a non-specific or generic face, having characteristics or featuresgenerally known to be part of faces. The model may correspond to, orform the basis of, a reference or prior knowledge for when imagesynthesis is to occur.

In one implementation, a suitable model can be a triangular mesh ofthree-dimensional points, specified by a set of vertices and edges.Alternatively, the three-dimensional model may be in the form of adiscrete cloud of three-dimensional points, from which a continuousthree-dimensional surface may be computed by standard methods ofinterpolation and extrapolation. In yet another implementation, themodel may be in the form of a deformable graph of connectedthree-dimensional points that can be perturbed in three-dimensionalspace in some specified way in order to account for geometric facialvariations among individuals or to represent facial expressions, such assmiling or frowning.

In step 220, the image that is to undergo recognition is analyzed todetermine pose and position of the object of interest. Given anon-frontal face photograph, one embodiment provides acomputer-implemented algorithm that analyzes the image in order torecover information indicating a rotation or position of the object inthe image relative to the camera. For example, with regard to facialrecognition, such an algorithm may first seek to recover the rotationand position of the face three-dimensionally, from the perspective ofthe camera lens. This can be formulated as the task of “virtually’orienting and positioning the three-dimensional model in space in such away that the simulated projection of rays from the model to the camerabest match the face in the photograph. For example, the ray departingfrom the tip of the nose of the three-dimensional model may intersectthe camera image plane at the tip of nose of the face in the photograph.The relationship between the points of the three-dimensional model inspace and the image points of the face in the photograph can bedescribed by a mathematical camera model, such as orthographic or fullperspective projection.

According to one embodiment, sub-step 224 (of step 220) provides thatattributes of position and orientation may be determined from use ofprojective three-dimensional models. In one embodiment, an unknownorientation and position of the object (such as a face) inthree-dimensional space can be specified in terms of a rotation matrix,and a translation vector. These unknown parameters can be computed byestablishing a set of correspondences between the three-dimensionalmodel points and the image points. For example, an image registrationalgorithm (such as described with one or more embodiments of FIG. 6) canidentify specific object or facial key-points in the image source 102.In the case where the object is a face, the points may correspond to thecorners of the eyes and mouth, and the tip of the nose. Since thepositions of these key-points on the three-dimensional model areestablished in advance, the unknown rotation matrix and translationvector can be computed by solving the system of equations resulting fromthe camera projection model. For example, closed-form, linearleast-square minimization methods or iterative, gradient descenttechniques can be applied for such task. In another embodiment, the poseestimation problem can be formulated as a classical vision trackingproblem, where optical flow computation between the original image andthe synthesized image is calculated in order to solve for pose andappearance at the same time.

Step 230 provides that the three-dimensional model (established in step210) is augmented using one or more attributes from the image. In oneembodiment, the attribute used to augment the three-dimensional model isintensity. As illustrated by sub-step 234, once the three-dimensionalorientation and position of the face has been recovered, an intensityvalue is associated to each point of the three-dimensional model. Theintensity value for each three-dimensional point is set to be the imagevalue at the projection of the three-dimensional point onto thephotograph, according to the recovered three-dimensional alignment andcamera model. This process augments the three-dimensional model with thedescription of the appearance of each model point in that photograph.

Some of the three-dimensional points of the model might not be visiblein the photograph due to occlusion caused by other parts of the face.The system can detect which points are not visible by using methods ofsurface-ray intersection computation. In one embodiment, the intensitiesof occluded three-dimensional points can be filled-in by exploiting facesymmetry: for example, the system might set the image intensities of theleft side of the nose to be identical to those of the right side, if theleft side is detected to be occluded.

As an alternative to face symmetry, one or more embodiments provide thatintensities of occluded regions can be marked, such as with text or aflag. For example, such portions may optionally be marked as “notavailable”.

In step 240, the three-dimensional model augmented with intensity valuesis used to synthesize the frontal view of the face. Thethree-dimensional model is first aligned to be in exact frontal positionand then standard rendering methods, such as ray casting, are used togenerate the new photograph.

According to an embodiment, if there are any occluded regions that aremarked as “not available”, the corresponding pixels are left unfilled inthe final photograph.

Programmatic and Intelligent Selection of Images for Training Sets andLabeling

In order to perform recognition of a desired object, image recognitiontechniques sometimes require a training set. The training set providesidentified faces (or other objects) that, as a point of comparison,provide identity information when recognition is performed on a set ofimages undergoing recognition. If an object in an image undergoingrecognition is deemed, through recognition, to be the same object (oralternatively the same class of object) as one of the objects in thetraining set, then the identity of the object under recognition isdetermined to be the same as that identity of the corresponding objectof the training set. In the case of facial recognition, for example, thetraining set may include images of faces of different persons. A set ofimages undergoing recognition may be compared against the training set,and if matches are found between the two sets, then the identity of arecognized face from the second set is assigned the identity of thematching face from the training set. In order to provide identityinformation, objects contained in images of the training set may containone or more signatures or other data identification, as well as a labelor tag provided by a user. For facial recognition, the label or tag maycorrespond to a name.

Embodiments described herein enable a user to scan a set of digitalimages in order to assign labels (e.g. corresponding identityinformation) to object or faces in the images. However, embodimentsdescribed herein also recognize that the user may not be the best sourcefor identifying the training set from a library of images. In manycases, users typically are unaware of what images in their data set orsuited for training set. For example, the user may identity imageshaving faces or objects that are difficult to recognize because, forexample, the object is severely tilted in its pose or positioned poorlyfor recognition in the image.

One or more embodiments provide for an algorithm that can be executedthrough use of a computer in order to programmatically select examplesof images that can provide an adequate reference for inclusion in atraining set. In one embodiment, an image for the training set may beprogrammatically selected by selecting the image of an object or facethat produces, from within a library of collection of images, themaximum recognized number of objects when used as part of training setby a face recognition algorithm. The programmatically selected images(or the portions containing the object that is to be recognized) may bedisplayed or otherwise rendered to the user. The user is then able tolabel the pictures, and the labeled pictures may be used as a trainingset by the image recognition program. An approach as described with suchan embodiment provides several benefits. The user is required to knowinformation for how to select images for a training set. An embodimentsuch as described may also optimize images that are returned and usedfor a training set, thus providing some limit to the amount of labelingthat may otherwise be required of the user. Moreover, the quality of thetraining set is high.

FIG. 3 illustrates a method for determining a set of training images foruse in performing image recognition for one or more objects of interest,according to an embodiment of the invention. A method such as describedin FIG. 3 may be used to establish a training set (or set of referenceimages) for use on a collection of images, and/or for newly received orcaptured images.

In a step 310, a set of candidate images are selected from a collection.Numerous techniques may be employed in order to determine an initial setof candidate images for a training set. The algorithm for choosing thetraining examples can be designed in many ways. In one embodiment, anyclustering algorithm can be used, such as k-means clustering. Theclustering algorithm would segment the group of faces into groups(clusters) based on a vector description (feature vector) for each face.Once the clusters are obtained, different metrics are obtained for eachcluster. These different metrics include, for example, cluster sizeand/or cluster compactness. Cluster size determines how many objects orfaces are included in that particular cluster. Cluster compactnessdetermines how spread the faces in that cluster are. The more spread isthe cluster, the more likely that there would be errors (multipleidentities) within that cluster. Metrics such as cluster size andcompactness may be used to choose cluster centers to be presented to theuser. The first metric optimizes the number of recognized faces, whereasthe second metric optimizes the accuracy of the recognition. Acombination of these metrics can also be used.

In step 320, the candidate images are presented to the user for labelingor other input. For example, for a given collection, a set of images maybe displayed to the user, where different faces or objects of interestare identified from the images in the set. According to oneimplementation, presenting the set of candidate images to the user isperformed with chosen cluster centers (corresponding to the face orobject closest to the cluster center).

In step 330, user-input is received. The user-input corresponds to, forexample, labeling. In facial recognition, for example, labeling maycorrespond to the user providing a name, nickname or other individual orclass identifier to a picture of a face or person.

In step 340, the training set is established using the candidate imagesand the input or labels provided from the user. The training set is thensubsequently used to recognize select or desired objects from imagesthat undergo recognition.

With regard to an embodiment of FIG. 3, it is also possible for a userto manually reject an image from a candidate set when the candidate setis presented to the user. For example, as described with an embodimentof FIG. 4, the candidate set of images may be displayed to the user, andthe user may make a correction or rejection to an image that contains aface that the user knows is mis-recognized, or otherwise not a goodbasis for use in recognition. As such, it is possible for the user toreject a candidate image.

As an alternative to an embodiment that uses clustering, anotherembodiment provides that step 310 may be performed, at least in part, byan algorithm that simulates face or object recognition. Such analgorithm may speculate different training sets, and then wouldprogrammatically select the training set that returns the maximumrecognized faces or objects. In one embodiment, such an algorithm may bedetermined by the following sub-steps:

(i) Obtaining a distance matrix, such that d_(ij) provides the distancebetween two faces i and j. This distance can also take the lighting, andpose of the face into account, and can apply additional penalty orweights for tilted pose, and directional lighting. In practice, thematrix can be sparse, and can be obtained using a nearest-neighborstructure. Instead of calculating for every i and j, the distance iscalculated only for the closest N faces in the nearest neighborstructure.

(ii) traversing the indexes from 1 to n, where for each index i, thefaces that are closer than a distance threshold, T (face recognitionacceptance threshold) are obtained. The list of indexes (CLOSELIST) andthe distance values are saved.

(iii) obtaining an optimization function over all faces. In oneimplementation, this function can combine many metrics including: (a)The number of faces that are identified to be closer than distance T insub-step (ii) (i.e. the size of the CLOSELIST{i} for every face i). (b)an aggregate sum (or the average) of the distances of faces that arecloser than distance T; and/or a combination of (a) and (b).

(iv) calculating the optimization function for every face i. A sortedindex is obtained by sorting these function values. This sorted list isthen traversed while a list of chosen labels and potentially recognizedfaces are constructed. The chosen labels list is the final list to beshown to the user, while the potentially recognized face list is thelist of faces estimated to be recognized given the selection of chosenlabels. A pseudo code for this step is provided below.

ChosenLabelList = empty list. PotentiallyRecognizedFaceList = emptylist. For j=1 to n  If SortedIndex(j) is not already inPotentiallyRecognizedFaceList,   put SortedIndex(j) in theChosenLabelList.   Put SortedIndex(j) in PotentiallyRecognizedFaceList.  Put all the faces in the CLOSELIST{SortedIndex (j)} into  PotentiallyRecognizedFaceList.  end endThe faces under the ChosenLabelList is shown to the user.

As described with one or more embodiments described above, labels maysubsequently be obtained from the user.

Object recognition systems and techniques (including general patternrecognition) may be implemented in a supervised or semi-supervisedlearning environment. Embodiments described herein further provide anadaptive acceptance threshold which controls the number of errors madeby the system during a learning process, based on the historicperformance on that task, as well as the expected difficulty of thetask.

FIG. 4 illustrates a semi-supervised process for dynamically altering anacceptance threshold for object recognition, according to one or moreembodiments of the invention.

In a step 410, a training set is formed. One or more embodiments providethat a training set may be formed using, for example, an embodiment suchas described with FIG. 3.

Step 420 provides that recognition is performed for individual images ina collection. The recognition may be for objects and/or faces. Therecognition may be performed by comparing objects of interest fromimages in a collection with corresponding images from the training set.In performing recognition of step 420, a sub-step 424 provides that aprogrammatic determination is made of individual images in thecollection that contain an object of interest, with confidence valuesthat within a designated threshold. The confidence values provide aquantitative measure as to a level of certainty that a recognition wascorrectly performed. Thus, for example, a program may scan and analyze acollection of digital images, and determine recognition for one or moreof the images, provided the confidence values of the recognition arewithin a designated threshold or tolerance level.

In a step 430, image results from performing a recognition process onindividual images contained in the collection are presented to the userfor verification and feedback. In particular, an embodiment providesthat the user may mark individual recognition results as correct orincorrect. Correct and incorrect results may be tabulated or otherwisetracked.

Step 440 provides that an error rate is determined from the userverification.

In step 450, the confidence value threshold for determining recognitionof objects in images is adjusted upward or downward, based on the errorrate. For example, if the error rate is determined to be high (based oninternal standard), the threshold may be lowered to reduce the errorrate.

As an alternative or addition to adjusting the threshold, correctlyrecognized results may be added to the training set, so that thetraining set is progressively increased in size. Likewise, an error setmay be maintained and updated for incorrect results. One or more both ofthe progressively updated training set and error set may be used toincrease the recognition rate.

The following provides an example of how an embodiment such as describedwith FIG. 4 may be implemented in the context of facial recognition. Auser supplies example photographs for use as part of a training set on acomputer on which image recognition software is made available. Theimage recognition software uses the training set to identify objects orfaces. When recognition is performed, the software may also produce aconfidence score for each object or face. The recognition processgenerates correct results as well as errors, which are undesirable. Anacceptance threshold is applied to the confidence score of each resultto determine which results to accept and which to reject. The user isthen presented with the accepted recognition results. The user may thenprovide input in the form of indicating whether individual results arecorrect or incorrect.

According to one embodiment, correct results may be added to thetraining set. Incorrect results may be added to a set of error examples.The set of error examples is used in turn along with the training setand potentially additional user supplied examples to run recognitionagain and increase the number of recognized faces. This process can goon for multiple iterations as long as the user supplies additionalinformation to the recognition software.

According to one or more embodiments, a user verification process isoften much more labor intensive when the user is presented with a lot oferrors. To maximize user satisfaction, the following parameters may becontrolled: (i) the total number of errors #E presented to the user; and(ii) the relative error rate, defined as #E/#R, where #R is the numberof results.

At the same time, the total number of correct faces recognized shouldnot be impacted.

Controlling the number of errors and the error rate can be achieved byaltering the acceptance threshold: a higher threshold will reduce thenumber of errors, at the cost of getting fewer recognition results. Alower threshold will cause the system to accept more recognitionresults, at the risk of letting through more unwanted errors. One ormore embodiments (such as described with an embodiment of FIG. 4)provide the threshold to be adaptive based on the expected number oferrors made by the system.

Since the number of errors #E made by the system is unknown at the stagewhere the threshold determination is made, and then only learnedsubsequently, one or more embodiments uses an estimate of #E todetermine the designated threshold on-the-fly. Such a determination maybe based on multiple sources of information, including but not limitedto:

1. the numbers #R and #E collected so far, that is: the number ofrecognized objects or faces and the number of errors made duringprevious iterations of the semi-supervised learning process. Thisinformation can be made available to the decision process by examiningthe size of the training set and the size of the error set.

2. the number #U of faces left to be recognized in the data, which is aclue to the expected #R in the next iteration.

More specifically, the decision threshold T can be written as:T=T0+f(#R, #E, #U), where T0 is a default threshold and f( ) a functionwhich maps the numbers collected from the data to a confidence penalty.

In a more specific instance, the expected error rate can be based on asmoothed estimate of the error rate so far: #EE=#E/(#R+a), ‘a’ being aconstant. The expected number of faces to be recognized can be based ona fraction of the total number of unknown faces: #UU=min(1, #U/b), ‘b’being a constant. The function f can then be a function of the twoparameters: f(#EE,#UU). As an example, f can increase with theproportional number of errors (#EE), but decreases with reduced numberof remaining unknown faces (#UU).

A graph of FIG. 5 shows a performance of one embodiment describedherein, on multiple datasets. The curves show the number of correctfaces recognized (Y axis) against the number of incorrect faces (X axis)after up to five iterations of the semi-supervised learning process.

The curves with ‘+’ markers use a fixed threshold T0, and the curveswith ‘.’ markers use the adaptive threshold T described in thisinvention. The dynamic threshold contributes to achieving variousbenefits, such as (i) not to reduce the number of correct facesrecognized, (ii) to control the number of errors made by the system (Xaxis), (iii) to control the relative error rate of the system.

In a graph shown, a relative error rate is the slope of the curve. As anillustration, the black diagonal line has a slope corresponding to a 10%relative error rate.

Registration

As mentioned, an image recognition system (such as for faces andobjects) can analyze two images and determine if the same subject (orclass of object) is present in both. To do this, regions aroundcorresponding points on the object of interest need to be compared.However, finding these corresponding points is a challenging task indigital images (such as consumer photographs), which contain largeappearance variability amongst faces or objects due to changes ofviewpoint, object deformations (such as expression on a face) andlighting. One or more embodiments provide a registration algorithm,which can find corresponding object features in the presence of suchappearance variability, as an integral part of an overall recognitionsystem for use in digital images.

Embodiments described herein include a system for finding correspondingpoints in objects of interest that appear in, for example, consumerphotographs. When applied to face registration, for example, such aprocess may be referred to as face registration. Given an image of aface, the algorithm determines the precise locations of a set ofmultiple features in the image, which include but are not limited to theeyes, the tip of the nose, the ends of the mouth, and the locations ofthe ears, hairline, and eyebrows. Similar processes may be performedwith other kinds of objects. For simplicity and illustrative purposes,however, some embodiments described herein may refer to techniques thatuse face registration, and it is understood that face registration isrepresentative of recognition of other objects.

In another embodiment, the locations of these marker features are usedto compute a dense correspondence map of between all pixels in the imageand the pixels of a generic face model, represented as a two-dimensionalimage. In another embodiment, the dense correspondence map can becomputed between the image pixels and the vertices of a genericthree-dimensional object or face model.

According to an embodiment, an algorithm for object or face registrationmay be configured to locate a location of multiple marker features in aphotograph. As mentioned, a photograph containing a face, for example,may have marker features corresponding to the eyes, mouth, nose etc butalso any other parts of the face which may not correspond to a specificanatomic part.

As mentioned with one or more embodiments, an object of interest maycorrespond to a face or some other object.

Accordingly, one or more embodiments define a probabilistic model in theform of a Markov network, which combines information about localappearance of the features in images, as well as geometric constraintson the relative location of the features. A Markov network associates alocation variable X_(i) with each face feature Fi. The domain of thevariable is discrete, corresponding to several distinct locationhypotheses for that feature in the image. A feature appearancepotentials may be defined asφ(Xi=[x,y])  (1)which estimate the likelihood that feature Fi is located at pixel [x,y]in the image based on the appearance of the image around that location.

In one embodiment, a local feature appearance model may be used. Such amodel may be learned from ground truth data, which contains exampleimages of the objects of interest, or faces, where the locations of thefeatures of interest are human-labeled. In one embodiment, a Gaussianappearance model is assumed for the pixel intensities of an image patchcentered on a particular feature. A low-dimensional feature space isestimated by using Principal Component Analysis (PCA) and the largestprincipal components are retained. The local appearance potentialφ(Xi=[00x,y]) assigns a high likelihood in cases when (i) The differencebetween the image patch centered on [00x,y] and its PCA spacereconstruction is small, and (ii) The Mahalanobis distance between themean feature appearance and the image patch projection to PCA space issmall.

The overall likelihood is determined as a weighted combination betweenthe above two criteria, where the tradeoff is estimated bycross-validation on the ground-truth data.

In another embodiment, a local feature appearance model can be learntusing classifiers such as boosting (Adaboost) or support vector machines(SVM). In this formulation, the ground truth data for each featurecontains a set of positive and negative examples, and the goal is toclassify the patch into one containing the feature at its center or not.A trained classifier (Adaboost or SVM) not only classifies a given patchbut also provides a certainty score. This certainty score can be used toobtain the values of the local appearance potential φ(Xi) as follows. Asubset of the ground truth data (a holdover set) is left out from theexamples provided to the classifier. The learned classifier is appliedto this subset of examples and for each certainty score of theclassifier the probability that the classifier is correct is estimated.To avoid overfitting, the ratio can be computed for certainty scoreranges. Alternatively, a logistic regression model can be used. Theresulting probabilities are used as values in the local appearancepotential φ(Xi).

The probabilistic model also enforces constraints on the relativelocation of the face features. This is done with pairwise potentialsφ(Xi, Xj) although higher-order potentials can be used as well. Suchpairwise potentials assign a probability for each possible combinationof the locations of Xi and Xj, expressing preferences such as “the righteye is placed to the right of the left eye”. They can be introducedbetween all pairs of features (O(N²) potentials for N features) orbetween each feature a few of its nearest neighbor features (complexityO(kN) if k neighbors are used). The values of these features areestimated from data as follows. The oriented vector d=Xj−Xi for eachconsidered pair of features is computed in all labeled trainingexamples, and a Gaussian distribution is learnt over the vectors d_(ij)for each pairwise potential φ(Xi, Xj).

The feature appearance and relative position in images changes fordifferent orientations of the object of interest. Therefore, one or moreembodiments recognize that it may be advantageous to group the groundtruth examples by face pose, and learn separate feature appearance andrelative feature location models for each of these poses. In anotherembodiment, when the appearance models are not substantially different,the same appearance model can be used for several features and/or inseveral poses (for example, the same appearance model can be used forright eye and left eye).

FIG. 6 illustrates a method for determining face or object registration,according to an embodiment of the invention. In step 610, an object ofinterest may be detected. In one embodiment, for example, theregistration algorithm may be performed for faces, and such an algorithmmay require face detection, as described in, for example, U.S. patentapplication Ser. No. 11/246,589, filed Oct. 7, 2005, and incorporated byreference herein. The result of face detection is the determination of aface box in which registration is to be performed, as well as thelocation of any two standard for marker features of the face. These twofeatures are used to scale and rotate the image appropriately beforeface registration is performed. Recognition of non-facial objects maysimilarly be performed.

In step 620, a set of location hypotheses for each object feature ormarker may be determined. In one implementation, a Markov network may beconstructed to contain single and pairwise potentials, φ(Xi) and φ(Xi,Xj), as described elsewhere in this application.

Step 630 provides that a most likely hypothesis location is determinedfrom the set for each of the markers or features of the object ofinterest. In one implementation, for example, a Loopy Belief Propagationmay be used to obtain the most likely feature locations that maximizethe probability defined by the Markov network. Because different Markovnetwork models have been learned for each different face pose, inprinciple, an algorithm may be implemented to try all these models andselect the model and solution, which provides the highest jointlikelihood.

For efficiency, one or more embodiments incorporate a pose classifier topredict likely face or object poses. The registration module then mayuse Markov network inference only for those poses. The pose classifiercan be trained using Adaboost, or can use separate Gaussian faceappearance models for each pose, similar to the local feature appearancemodels described elsewhere in this application.

The feature or marker locations of the face or object of interest,obtained as described with an embodiment of FIG. 6, can be used tocompute a dense correspondence map between the face image and a generictwo-dimensional face model. Given the locations for the features, adeformable mapping for the remaining pixels of the image can beestimated. An affine mapping or thin-plate spline models, among others,can be used. All images mapped to the same generic model can be easilycompared.

In another embodiment, the feature locations can be used to compute adense correspondence map between the face image and a genericthree-dimensional face model. Again, images mapped to the samethree-dimensional face model can be easily compared.

FIG. 7 illustrates an image recognition system constructed according toembodiments described herein. In embodiment, a system 700 includesobject detection module 710, registration module 720, synthesis module730, training module 740, and recognition module 750. The objectdetection module 710 may, as described with one or more embodimentsprovided herein, detect a region of an image where a face or otherobject of interest is located. An input for the object detection module710 is an image file 708 or other image source. The output of the objectdetection module 710 is image data for a region of an image thatincludes the detected object,

The registration module 720 performs a registration process or algorithmon the region of the image where the object of interest is detected, asprovided by detected object image data 712. The registration module 720may implement one or more algorithms for performing registration,including, for example, an algorithm in accordance with an embodiment ofFIG. 6. The registration module 720 may provide a registered imageoutput 722, for use by the synthesizer module 730 and/or recognitionmodule 750.

If the object is posed at a tilt that makes complete recognitiondifficult or impossible, the synthesizer module 730 may be used togenerate or synthesize an image, provided by synthesized image data 732.The synthesized image data 732 may accommodate or compensate forpresence of occlusion or distortion, as a result of the tilt in theobject of interest in the image file 708. As an alternative or addition,the image may be normalized to facilitate the recognition. Thesynthesizer module 730 may be configured according to, for example,using a system such as described with an embodiment of FIG. 1A, and/orotherwise be configured to implement an algorithm such as described withan embodiment of FIG. 2.

The recognition module 750 may use synthesized (for tilted posedobjects) and non-synthesized images (when appropriate or possible) or acombination of the two to perform recognition. A reference or trainingset 742 may be established for use by the recognition module 750.Training module 740 may be configured, for example, in accordance withan embodiment of FIG. 3 and/or FIG. 4. As such, training module 740 maycontinuously gather images from a collection or library of images 744for purpose of refining or increasing the training set 742. The trainingset 742, and labels associated with individual images, enablesrecognition module 750 to provide a desired output (e.g. name or otherlabel identification) when the recognition is performed. As a result ofimplementing the training module 740 such as described by embodimentsherein, recognition module 750 may provide better results.

Moreover, as described above, one or more embodiments provide that thetraining set 742 created by the training module 740 becomesprogressively larger based on an error rate, further increasing therecognition module's performance. The use of synthesizer module 730, toprovide synthesized image data 732 when necessary, also enhancesperformance of the system as a whole, as recognition can be carried outto objects with tilts and poses that would otherwise be too severe forregular recognition.

FIG. 8 illustrates a computer system on which embodiments such asdescribed with FIGS. 1-7, and combinations thereof, may be implemented.A system for implementing one or more embodiments such as described mayinclude a computer 800, such as a desktop computer, work station, oreven server. Processing resources 810 may implement the algorithmsdescribed herein. A memory resource 820 including data structures 824may store image files 832 and/or data for identifying locating imagefiles on other computers. The computer 800 may include one or morecommunication ports 805 for extending communications across a network802. The memory resource 820 and/or data structures 824 may also storeinformation 834 associated with images that are to be recognized, andinformation for images that are recognized. Examples of the types ofinformation that may be stored on the computer 800 include image files(and/or information for identifying locations of image files),information identifying training sets, labels associated with imagefiles of the training set, from which recognition occurs, andaugmentation information 134 (see FIG. 1). The computer 800 may beoperated locally, or from a network. Moreover, some or all of theinformation determined or stored on the computer 800 may be local orstored on a network. Still further, some information or programs oralgorithms may be performed external to the computer 800, and resultsfrom those programs or processes may be shared with the computer 800.Additionally, the computer 800 may be part of shared computingenvironment (through use of the network 802 or otherwise) for performingany one of the algorithms, processes or functions described herein.

In an embodiment, computer 800 may be part of a server system thatprovides a web service. For example, computer 800 may store image filesfrom users that communicate with the computer using an Internetconnection and web browser. In one implementation, computer 800 performsacts of recognition and rendering of results, as well as display ofuser-interfaces for receiving information such as labeling and responsesto programmatic training set determinations.

As mentioned, it is contemplated for embodiments of the invention toextend to individual elements and concepts described herein,independently of other concepts, ideas or system, as well as forembodiments to include combinations of elements recited anywhere in thisapplication. Although illustrative embodiments of the invention havebeen described in detail herein with reference to the accompanyingdrawings, it is to be understood that the invention is not limited tothose precise embodiments. As such, many modifications and variationswill be apparent to practitioners skilled in this art. Accordingly, itis intended that the scope of the invention be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anembodiment can be combined with other individually described features,or parts of other embodiments, even if the other features andembodiments make no mentioned of the particular feature. This, theabsence of describing combinations should not preclude the inventor fromclaiming rights to such combinations.

What is claimed is:
 1. A computer implemented method comprising:identifying an object in an image; obtaining, from a database ofpredetermined object models, an object model for the object identifiedin the image, the object model comprising a plurality of threedimensional points from which a three dimensional surface may becomputed; determining, using the image and the object model, anorientation and one or more occluded features of the object in theimage; augmenting the object model to compensate for the one or moreoccluded features of the object in the image, wherein augmenting theobject model comprises: augmenting an intensity value associated with atleast one of the plurality of three dimensional points; and rotating theobject model; and providing the augmented object model as output.
 2. Themethod of claim 1, wherein identifying an object in an image comprises:identifying one or more image markers included in the image, each imagemarker comprising one or more predetermined image features; determiningthat at least some of the one or more image markers match predefinedmarkers of the object model; and identifying the object in the image asan object represented by the object model.
 3. The method of claim 2,wherein determining an orientation of the object in the image comprises:determining, for each of the one or more image markers, a relationshipbetween the image marker and a matching predefined marker matched by theimage marker; and using each relationship to determine the orientationof the object in the image.
 4. The method of claim 1, wherein augmentingan intensity value associated with at least one of the plurality ofthree dimensional points comprises: using one or more intensity valuesof points in the image to augment intensity values of respective threedimensional points.
 5. The method of claim 1, further comprising: priorto augmenting the object model, orienting the object model to pose theobject model at an orientation that matches the orientation of theobject in the image.
 6. The method of claim 1, wherein rotating theobject model comprises rotating the object model to a predefinedorientation.
 7. The method of claim 1, wherein providing the augmentedobject model as output comprises providing the augmented object model asoutput to an object recognition system.
 8. A system comprising: one ormore data processing apparatus; and a data storage device storinginstructions that, when executed by the one or more data processingapparatus, cause the one or more data processing apparatus to performoperations comprising: identifying an object in an image; obtaining,from a database of predetermined object models, an object model for theobject identified in the image, the object model comprising a pluralityof three dimensional points from which a three dimensional surface maybe computed; determining, using the image and the object model, anorientation and one or more occluded features of the object in theimage; augmenting the object model to compensate for the one or moreoccluded features of the object in the image, wherein augmenting theobject model comprises: augmenting an intensity value associated with atleast one of the plurality of three dimensional points; and rotating theobject model; and providing the augmented object model as output.
 9. Thesystem of claim 8, wherein identifying an object in an image comprises:identifying one or more image markers included in the image, each imagemarker comprising one or more predetermined image features; determiningthat at least some of the one or more image markers match predefinedmarkers of the object model; and identifying the object in the image asan object represented by the object model.
 10. The system of claim 9,wherein determining an orientation of the object in the image comprises:determining, for each of the one or more image markers, a relationshipbetween the image marker and a matching predefined marker matched by theimage marker; and using each relationship to determine the orientationof the object in the image.
 11. The system of claim 8, whereinaugmenting an intensity value associated with at least one of theplurality of three dimensional points comprises: using one or moreintensity values of points in the image to augment intensity values ofrespective three dimensional points.
 12. The system of claim 8, whereinthe operations further comprise: prior to augmenting the object model,orienting the object model to pose the object model at an orientationthat matches the orientation of the object in the image.
 13. The systemof claim 8, wherein rotating the object model comprises rotating theobject model to a predefined orientation.
 14. The system of claim 8,wherein providing the augmented object model as output comprisesproviding the augmented object model as output to an object recognitionsystem.
 15. A non-transitory computer readable medium storinginstructions that, when executed by one or more data processingapparatus, cause the one or more data processing apparatus to performoperations comprising: identifying an object in an image; obtaining,from a database of predetermined object models, an object model for theobject identified in the image, the object model comprising a pluralityof three dimensional points from which a three dimensional surface maybe computed; determining, using the image and the object model, anorientation and one or more occluded features of the object in theimage; augmenting the object model to compensate for the one or moreoccluded features of the object in the image, wherein augmenting theobject model comprises: augmenting an intensity value associated with atleast one of the plurality of three dimensional points; and rotating theobject model; and providing the augmented object model as output. 16.The non-transitory computer readable medium of claim 15, whereinidentifying an object in an image comprises: identifying one or moreimage markers included in the image, each image marker comprising one ormore predetermined image features; determining that at least some of theone or more image markers match predefined markers of the object model;and identifying the object in the image as an object represented by theobject model.
 17. The non-transitory computer readable medium of claim16, wherein determining an orientation of the object in the imagecomprises: determining, for each of the one or more image markers, arelationship between the image marker and a matching predefined markermatched by the image marker; and using each relationship to determinethe orientation of the object in the image.
 18. The non-transitorycomputer readable medium of claim 15, wherein augmenting an intensityvalue associated with at least one of the plurality of three dimensionalpoints comprises: using one or more intensity values of points in theimage to augment intensity values of respective three dimensionalpoints.
 19. The non-transitory computer readable medium of claim 15,wherein the operations further comprise: prior to augmenting the objectmodel, orienting the object model to pose the object model at anorientation that matches the orientation of the object in the image. 20.The method of claim 1, wherein determining the one or more occludedfeatures of the object in the image comprises identifying one or moresurface-ray intersections in the image.